fix(rollout): train safely on incomplete groups - #6

…roject#1569)

agenerate read the weight version twice: once when building the
request and again when recording output_versions. A weight update
landing in between labelled tokens with a version that did not
generate them, which skews staleness checks such as the decoupled
loss and rejection sampling.

Pin the version before the request goes out and reuse it when
recording, so each segment of a trajectory carries the version that
actually served it.

…real-project#1544)

use_deterministic_algorithms previously set deterministic_mode only on
the built model's config. Several Megatron-Core and TransformerEngine
code paths consume determinism settings at module construction and
cache them in instance state:

- VocabParallelEmbedding copies config.deterministic_mode at __init__;
  without it the nondeterministic F.embedding backward is used.
- TEDotProductAttention validates NVTE_ALLOW_NONDETERMINISTIC_ALGO
  against deterministic_mode only at __init__, and TE snapshots its
  deterministic flag from that env var and the global torch switch at
  construction.

Setting the flag after the model was built therefore engaged only
runtime consumers (loss fusions, schedules) and silently left the
layer-level kernels nondeterministic.

Changes:
- Set deterministic_mode on the TransformerConfig before the model is
  built, keeping the post-build call for runtime consumers.
- Select AttnBackend.flash under deterministic mode: Megatron-Core owns
  the NVTE_*_ATTN selection env vars and asserts they match the config,
  and the cuDNN fused-attention deterministic backward needs workspaces
  that grow prohibitively with context length.
- Export NVTE_ALLOW_NONDETERMINISTIC_ALGO=0 to trainer processes from
  the launchers and to megatron worker specs in single-controller mode,
  so it is in place before TransformerEngine reads it regardless of TE
  version; warn when the setting may have come too late.

With this, repeated runs of the same batch produce bitwise-identical
training stats, including grad_norm.

…(incl. areal-project#1577) (areal-project#1579)

* fix: emit PEFT-standard disk LoRA adapter keys so vLLM can load them, and align sglang best-effort unload test assertion (areal-project#1577)

* test(inference-service): raise VLM controller init timeout to 600s to avoid vLLM startup timeout

…real-project#1573)

Upstream SGLang does not enable the mamba cache path for the Bailing
hybrid MoE family, so radix cache stays off and prefix reuse is lost.
Add an opt-in script that applies the same eight edits the Ling hybrid
fork carries, covering SGLang 0.5.9 and 0.5.10.post1, which moved the
DP-attention padding helper behind a dp_size prologue.

The script is reversible and supports a dry run, so the target tree can
be checked before anything is written.

…ndering (areal-project#1499)

* fix(openai): align proxy tool schemas with sglang chat-completions rendering

The proxy path renders the request `tools` block through LiteLLM-style
dicts, while sglang's native /v1/chat/completions route round-trips
tools through its pydantic `Tool` model before applying the chat
template. The two renderings differ in field order and default-field
presence (e.g. sglang's `Function` dumps `strict: false` while the
proxy omits it), so the same trajectory produces different prompt
token ids on the two paths and training rewards drift from evaluation
results.

Round-trip tools through sglang's `Tool` model on the proxy path so
the dicts fed to `apply_chat_template` byte-match sglang's own
rendering. Flat Responses `FunctionToolParam` entries (top-level
`name`/`parameters`, no `function` key) are first normalized to the
nested Chat shape that both sglang's model and the chat template
expect. The alignment only runs when the engine class name identifies
an sglang backend; other backends are left unaligned, and per-tool
validation failures fall back to the original dict.

Ported from a verified internal fix.

* test(openai): cover tool schema alignment against sglang rendering

Assert the aligned dicts match what sglang's own Tool model produces, so
a regression that reintroduces the prompt-token drift fails here rather
than showing up as a train/eval reward mismatch.

Also pin the documented fallbacks: flat Responses tools normalize to the
nested Chat shape, a single invalid tool does not fail the batch, and
unsupported entries pass through unchanged.

)

* feat(mcore): apply fp32 lm head forward when enabled

enable_fp32_lm_head only reached the model through mbridge extra args, so
it was dropped for configs whose TransformerConfig rejects the field and
the flag silently had no effect. Patch the lm_head/output_layer forward
after the model is built instead, so the existing flag applies on every
Megatron construction path.

The patch is skipped when the flag is off, when Megatron already provides
a native fp32 column-parallel head, and when a module was already
patched, and it never runs for critic models, which replace the output
layer with a value head.

* fix(mcore): forward the tensor-parallel group from the fp32 lm head

The patched forward mirrors ColumnParallelLinear, which passes self.tp_group to
copy_to_tensor_model_parallel_region and
gather_from_tensor_model_parallel_region. Both calls omitted it, so they fell
back to the default tensor-model-parallel group and ran the collectives on the
wrong ranks whenever the head was built with a non-default group. The sequence
parallel path already forwarded the group, so the two behaved inconsistently.

Take the group once and route all three calls through a helper that drops the
keyword on megatron-core releases that do not accept it. Runs on the default
group are unaffected: the wrappers resolve None through
get_tensor_model_parallel_group_if_none, so passing it explicitly is identical
to omitting it.

Document the patch on _fp32_lm_head_forward_impl: what it fixes, the
megatron-core version it was verified against, and how it drifts if upstream
changes the forward or the collective signatures.

…l-project#1572)

Rejection sampling narrows the loss mask, so the existing token counters no
longer describe what the update actually trained on. Report the total, valid and
masked token counts alongside the masked ratio, and split the log-prob drift
into signed and absolute forms so a run's staleness is visible per step.

prompt_len is derived from the first trained position rather than the difference
of the two mask sums, which stops reporting the prompt as longer than it is once
rejection removes generated tokens from the loss mask. The mask reaching
_ppo_update is rolled left by one, so the roll is undone before the lookup.

Address review on estimator-owned group minimums and the tightened
workflow contract:

- NormConfig gains uses_group_statistics and PPOActorConfig owns
  minimum_usable_group_size, replacing the trainer-side free function
  and its ad-hoc shape duck-typing.
- A singleton target group is complete by definition, so group
  statistics no longer hard-fail n_samples=1 configs; the previous
  ValueError broke examples/openclaw on startup while the v2 rollout
  path skipped the check entirely. Config-level degeneracy stays a
  UserWarning, as on main.
- The trainer collection loop shares the empty-round streak plus
  wall-clock stall bound and rate-limits its progress warnings.
- WorkflowContractError now names the two remedies (one sample per
  episode or batch-level normalization), and the one-sample-per-slot
  contract is documented in the RolloutWorkflow docstring and the
  grouped-rollout reference (EN/ZH), scoped honestly to grouped
  rollouts: n_samples=1 installs no group wrapper and is not checked.
- Correct docstrings that still described divisibility-based dispatch:
  balanced_greedy_partition, _pad_eval_batch, and the v2 data-proxy
  dispatcher module.

Refs: areal-project#1563
Signed-off-by: EazyReal <8047065+EazyReal@users.noreply.github.com>

With n_samples=1 every prompt group is a singleton, so group mean
centering erases the task reward. Batch centering keeps a live signal;
singleton group std is already pinned to 1 and stays a no-op.

Signed-off-by: Max Wang <maxwill@vmax.ai>

…e-rollout-groups

# Conflicts:
#	areal/trainer/ppo/actor.py

Reviewer request on areal-project#1563: an explicit min_usable_group_size below 2
combined with group-relative normalization would train lone survivors
that have no group peers to normalize against, so PPOActorConfig now
rejects it at construction. The derived default is unaffected.

Signed-off-by: Max Wang <maxwill@vmax.ai>

std_level: group is not the no-op I claimed on the review thread: the
OpenAI-proxy agent exports one row per interaction, so compute_advantages
passes per-episode row counts as group sizes and a k-turn episode's
identical rewards collapse to sign(r - mean) * sqrt((k-1)/k), discarding
reward magnitude. Batch std matches adv_norm and keeps the signal; the
group_size line had no remaining consumer.

Signed-off-by: Max Wang <maxwill@vmax.ai>

Two exactness gaps around the new config field. The v2 rollout path never
consumes it, so an explicit setting now fails fast there (matching
RolloutControllerV2's rejection of reward_normalization and
drop_incomplete_group) and the help text names the v1 scope. The
one-sample-per-slot contract is armed by the resolved minimum, not by
group normalization itself, so the error message, arun_episode docstring,
and reference docs now name the real trigger and include unsetting
actor.min_usable_group_size among the remedies; the agent tutorial and
export-style reference gain the same caveat where they teach
export_style: individual.

Signed-off-by: Max Wang <maxwill@vmax.ai>

* feat: update awex to 0.8.0

Bump the pinned awex weight-synchronization dependency in both the
SGLang and vLLM manifests and regenerate the corresponding lockfiles.

awex 0.8.0 adds Qwen3-MoE weight conversion with dtype-only IPC
grouping and an SGLang colocate plugin with colocate P2P transfer
fixes. It also corrects the fused qkv split to be GQA-aware and forces
CUDA IPC handles to close before signalling the train side.

All 25 awex symbols imported by AReaL were audited against v0.8.0 and
are unchanged in both name and signature, so no call-site changes are
required. awex declares no runtime dependencies, so resolution is
otherwise unaffected.

Key changes:
- Pin awex==0.8.0 in pyproject.toml and pyproject.vllm.toml
- Regenerate uv.lock and uv.vllm.lock

* chore: update Dockerfile node version and lock npm version

…ct#1596)

Use a Gloo sidecar for separated AWEX liveness and completion barriers so the NCCL process group remains dedicated to model-weight payloads. Fence SGLang device writes before the CPU-side completion barrier so success is not reported while CUDA copies are still pending.

Key changes:
- create matching NCCL and Gloo groups for FSDP, Megatron, and SGLang adapters
- route setup and completion barriers through the sidecar
- synchronize SGLang device copies before the completion barrier
- destroy both groups and test lifecycle and ordering

…ect#1589)

Preserve model-declared worker order regardless of registration timing, and prevent concurrent SGLang servers from sharing Triton cache entries.

Key changes:
- Rebuild router candidates in model address order
- Deduplicate model addresses without changing routing weight
- Give every SGLang fork an index- and UUID-scoped cache directory
- Preserve existing controller-to-worker cache root precedence

…on Slurm (areal-project#1584)

SchedulingSpec had no way to reach sbatch's --reservation or --exclusive, so
runs that need a reserved partition had to be launched by hand outside the
scheduler.

Explicit env_vars were also lost: the scheduler applied AReaL's forwarding and
the thread-count defaults on top of them, so a spec asking for a specific
OMP_NUM_THREADS or allocator config silently got the framework value. Snapshot
the user's mapping and re-apply it last, which also lets colocated roles carry
per-role settings that must differ between actor and rollout.

Allow optimizer configs to select an absolute scheduler warmup while preserving the proportional fallback across FSDP, Megatron, and Archon.

Validate shared scheduler boundaries and keep Megatron resume initialization independent of the active warmup config.

Key changes:
- Add fixed warmup resolution with cross-engine validation
- Align Megatron decay and resume scheduler parameters
- Add boundary and Megatron integration regression tests

Co-authored-by: 峯回 <dh183333@antgroup.com>

…-project#1602)

The normalization section still claimed that `group_size` must be
specified for group-level normalization. Since areal-project#1454, group boundaries
come from the rollout batch metadata (`TrajBatchMeta.traj_group_sizes`)
that `PPOActor._compute_advantages` forwards to `Normalization`, and
`NormConfig.group_size` only serves as the fixed-stride fallback when
that metadata is absent. Mirrored in the Chinese translation.

Co-authored-by: EazyReal <8047065+EazyReal@users.noreply.github.com>

…t#1594)

* fix(models): support FP32 operands with chunked LM head

Keep enable_fp32_lm_head orthogonal to chunked logits while avoiding repeated local vocab-weight casts across sequence chunks.

Key changes:
- Reuse one FP32 weight conversion per LM-head forward and backward
- Preserve TP/SP gradient communication and FP32 main_grad accumulation
- Cover flag combinations, gradients, and distributed execution

* test(models): account for chunked bias reduction order

Low-precision chunked backward sums bias gradients per chunk, while the full reference reduces all tokens at once. Compare the old BF16/FP16 path with an explicit relative tolerance and keep strict parity for FP32 operands.

* refactor(models): scope FP32 projection helper

Keep the full-sequence FP32 projection implementation private to the native linear class so unrelated call sites cannot invoke it accidentally.

…al-project#1575)

* feat(scheduler): support grouped colocation in the Ray scheduler

Colocation previously required the colocated role to match the target
role's replica count, which rules out AWEX-style colocation where a few
multi-GPU inference workers share the GPUs of many single-GPU trainer
workers (e.g. 16 x 4-GPU SGLang servers over 64 x 1-GPU actors).

When replica counts differ but the colocated role's total GPU demand
exactly reuses the target role's GPUs, route worker creation to a
grouped path: chunk each target node's physical GPUs into contiguous
per-worker groups (never crossing nodes), pin one zero-GPU process
launcher to each target node via node affinity, and start worker
processes with explicit physical gpu_devices so CUDA_VISIBLE_DEVICES
carries physical indices exactly like the Slurm launcher. Grouped roles
own their workers and launchers, are not registered as colocated
aliases, and create no placement groups, so readiness discovery and
teardown follow the standard paths.

* fix(scheduler): accept a device-free Ray driver node

RayScheduler probed accelerators on the driver process only, so a CPU-only
head node aborted with "does not support CPU-only clusters" even when every
Ray worker exposed GPUs. Fall back to the advertised cluster resources when
the driver itself owns no device.

…l-project#1603)

Megatron-Bridge materializes a provider separate from the transformer config. Apply prebuild determinism to that provider before finalization so construction-time consumers inherit the requested settings.

Avoid wrapping local targets with stdbuf when the effective target
environment already defines LD_PRELOAD. GNU stdbuf appends libstdbuf to
the value, which breaks TMS CDLL loading during offload.

Preserve env=None versus explicit env={} semantics while retaining line
buffering for targets without LD_PRELOAD.

Refs: areal-project#1570

Signed-off-by: yaoyaoshiguonan <yaoyaoguonan@outlook.com>

The no_save_optim help text told users the flag was "required when
using use_distributed_optimizer with Megatron (flattened_range
incompatibility)". That workaround was obsoleted by areal-project#1468, which
switched the Megatron checkpointer to dp_reshardable optimizer
sharding, so saving optimizer state works again on the pinned
megatron-core 0.17.0.

Following the stale advice now costs users silently: recovery resumes
with a freshly initialized optimizer, continuing at full learning rate
with reset Adam moments. Reword both optimizer-skip flags to describe
the actual trade-off, and add tests pinning the behavior the help text
now promises.

Key changes:
- Reword no_save_optim/no_load_optim help in RecoverConfig
- Assert both flags default to off, guarding against reintroducing the
  workaround as a default
- Assert the help text no longer documents the flags as required
- Cover RecoverHandler threading both flags into SaveLoadMeta.with_optim
- Regenerate docs/{en,zh}/cli_reference.md

Refs: areal-project#1341, areal-project#1468

Allow Qwen3-VL models to merge multimodal embeddings before packing while preserving existing padded-only and wrapper-owned paths.

Key changes:
- Route dense and MoE Qwen3-VL through model-owned THD
- Reconstruct padded inputs and restore packed model outputs
- Add routing, alignment, parity, and distributed forward coverage

…ct#1605)

* feat(engine): support Qwen3-VL with native AWEX colocate

Preserve the complete multimodal Hugging Face config during colocate\nmetadata exchange so AWEX can resolve both the language model and vision\ntower sharding contracts.\n\nKey changes:\n- Publish composite Qwen3-VL inference configuration\n- Delegate nested config parsing to native AWEX\n- Add Dense and MoE colocate contract tests

* fix(engine): preserve Qwen3-VL-MoE composite config

SGLang stores only the text config on the Qwen3-VL-MoE runtime model.\nRead the original Hugging Face config from ModelRunner so AWEX also\nreceives the vision tower metadata required for weight sharding.

…1617)

Point Ascend users to the ascend-v1.0.5 branch and refresh the guide to match the published A2 and A3 images. Document HDK 25.5.1, CANN 9.0.1, and the verified runtime stack, and keep the Chinese translation aligned with the English guide.

…1618)

Proxy engines keep issuing stale requests during rollout weight updates.
Pause them before normal workers and resume them before dispatch restarts.

* feat: add turn-level GAE support

Treat each generated turn as a GAE timestep while preserving the token-level default and token-local KL regularization.

Key changes:
- propagate and validate token-aligned turn IDs
- compute turn-level advantages without full CPU sequence scans
- filter structural metadata at FSDP and Archon model boundaries
- document the new selector and add focused regression tests

* feat(trainer): add dynamic per-sample GAE lambda

Allow GAE lambda to vary by trajectory using effective token or turn lengths while preserving static float behavior.

Key changes:
- Resolve custom lambda functions and keyword arguments from config
- Add VAPO length-adaptive GAE with empty-trajectory handling
- Validate per-sample lambda tensors and cover token and turn modes

* perf(trainer): reduce GAE preprocessing overhead

Hoist loop-invariant tensor work and bypass dynamic trajectory length construction when GAE lambda is static.

* feat(trainer): add relative-position GAE lambda

* fix(trainer): allow token lambda without turn metadata

Keep custom token-level GAE lambda functions compatible with rollout workflows that do not emit turn IDs, while preserving the metadata requirement for turn-level GAE.

* docs: regenerate CLI reference for GAE options

Keep the generated configuration reference aligned with the current main branch after porting the AntCode GAE changes.

* docs: document flexible GAE configuration

Explain token- and turn-level recurrences, KL and critic semantics, dynamic lambda strategies, and custom workflow turn IDs in English and Chinese.\n\nFix CLI default rendering for dataclass factories and cover it with unit tests.

* refactor(trainer): extract GAE helpers

Keep PPOActor focused on training orchestration by moving GAE kernels, turn metadata validation, and lambda context construction into a dedicated module.

---------

Co-authored-by: Wenhao Zhou <miumiu.zwh@antgroup.com>

Enable Qwen3.5 Dense and MoE registry recovery for AWEX colocated weight updates and provide a two-node Megatron/SGLang Geometry3K configuration.

…1601)

Keep prior versioned LoRA names registered in the vLLM server so
in-flight rollout requests do not break after adapter refreshes.
Also fixed lora example yamls as per some latest syntax.

Key changes:

- Retain old runtime LoRA aliases in areal_vllm_server.py
- Add a regression test for alias retention

…rollout-groups